CamelCamelCamel's Communication with the Amazon Service
Understand communication between CamelCamelCamel and Amazon services.
Introduction#
The C3's services interact with Amazon's services to provide a seamless user experience. In this lesson, we’ll see how this interaction happens in the following two scenarios:
C3 service communicating client requests to Amazon.
C3 service communicating server requests (C3 requests) to Amazon.
In the case of client requests, the C3 service prepares a new request based on queries received in the client's request, sends it to the Amazon service, and waits for the response. After receiving a response from Amazon, the C3 service processes it and sends it to the client.
In the case of server requests, the requests are prepared based on generalized queries to get maximum data from Amazon. Let's elaborate on communication between C3 and Amazon in the following sections.
Search service#
Whenever a search request arrives from the client, the C3 service lets the client wait until it receives up-to-date results from Amazon's search service. The illustration below presents the complete flow of a search request by a client:
1 of 11
2 of 11
3 of 11
4 of 11
5 of 11
6 of 11
7 of 11
8 of 11
9 of 11
10 of 11
11 of 11
We already discussed the search message's format and endpoints between a client and service provider (the C3 service in our case). Let's discuss the format of the message from the C3 service and the endpoint, which communicates with the Amazon service in the upcoming section.
Base URL of Amazon service#
The following base URL is used to send requests to Amazon's services:
The following endpoint is used to perform searching and fetching product operations:
Message format for the search#
The data entities are the same as discussed in C3's service. We'll analyze the request and response format now:
HTTP method: To search for a product, the HTTP
POSTmethod fetches a list of available products based on the information passed in payload from the Amazon service.Request format: The request format for the HTTP
POSTmethod is defined below as given in the product advertising API's request by Amazon:
Amazon's authentication#
Amazon uses unique headers that are defined below to authenticate each request:
x-amz-date: 20160925T120000Z: A custom date header specifies the request's date and time to prevent third parties from intercepting the request for later submission. A request must reach the AWS server within five minutes of the timestamp defined in the request; otherwise, it’s denied by AWS. Moreover, this header is used to create a signature that’s compared later with a signature generated by the client in the authorization header.
x-amz-target: com.amazon.paapi5.v1.ProductAdvertisingAPIv1.SearchItems: This defines the targeted service’s endpoint to place the request. In our case, it’s `SearchItems`. This value eases the API gateway’s process so that it can send requests to the targeted service, and it is also used to create the signature.
Authorization: The authorization header consists of the following parameters:
AWS4-HMAC-SHA256 defines an algorithm used for signatures and reads as signature version 4 with the HMAC-SHA256 hash algorithm.
Credential=AKIAIOSFODNN7EXAMPLE is the secret access key assigned to authorized users.
SignedHeaders=content-type;host;x-amz-date;x-amz-target defines which headers are used to create the signature.
Signature=&5d672d79c15b13162d9279b0855cfba6789a8edb4c82c400e06b5924a6f2b5d7 is the signature value calculated using the hash algorithm with the defined headers.
The need for a signature helps to secure the requests by verifying the requestor's identity with a valid access key. To prevent request tampering during transit, the hash is calculated again using all the headers and verified with the one in the request. If they both do not match, the request is denied.
Response format: If the request is successful, the client receives the following response:
Product advertising service#
The functionality of the product advertising service is a bit different than a typical service provider. The C3 service sends several requests (server-sent requests) to Amazon's product advertising service to continuously get up-to-date data. These requests are periodic and are sent using the SQS service of Amazon.
Point to Ponder
Question
Why does the C3 service need to send several requests to Amazon’s product advertising service?
The product advertising service provides data about all the products on Amazon. The data can vary as soon as a vendor updates the specifications or price of a product. The C3 service sends several requests so that those requests are processed periodically (using SQS) and can get updated details about the products. Those updated details are saved in C3’s database and used to fulfill various functional requirements listed below:
- It suggests popular deals.
- Product service publishes an event of the price drop on the pub-sub after data analysis.
- It uses data as a backup to respond to user’s search requests in case the Amazon service becomes unresponsive (for some reason).
- Users can get the price timeline of the products.
Note: These are server-sent requests that are transparent to clients. The server sends these requests to maintain its local database, similar to how Amazon’s product database does maintenance.
C3's product service fills the SQS queue based on the allowed number of requests in Amazon's product service. The SQS queue sends each request to Amazon's product service periodically on behalf of C3's product service. The response (containing information on multiple items) from Amazon is returned to C3's product service, which is saved locally in C3's database. The SQS queue is filled again when the last request is fulfilled. This periodic pattern continues, keeping throttling in mind. The following illustration shows the workflow:
1 of 6
2 of 6
3 of 6
4 of 6
5 of 6
6 of 6
Note: SQS queue is filled with requests (due to rate limiting per day or hour) that are sent to the PA-API periodically. Each request’s response contains multiple items processed and saved in the C3's database. The queue is emptied after a while because the rate limiter allows our requests to pass through because of a lull period after the previous burst. After the queue is emptied, it is filled again with requests, and this process continues. It sends requests in multiples to reduce the burden of C3's product service instead of sending them individually to the SQS queue.
In the following section, let's discuss the format for each request sent to the SQS queue.
Message format for the product advertising service#
To get details about products from the product advertising service, the C3 service will interact with PA-API. We’ll target the GetItems endpoint to get details of the product based on productID (ASIN in the case of Amazon's products). We’ll analyze the request and response formats that interact with PA-API below:
HTTP method: We’ll use the
POSTmethod to fetch results from a server.Request format: The request format defined by Amazon for HTTP
POSTmethod is defined below:
Response format: If the request is successful, the following response is sent back to the C3 service:
In functional requirements, we mentioned that clients could add any product to the watch list and get notified when prices drop. Based on the user's requirements, the C3 service analyzes fetched data (presented above in the response). If a product’s price meets the user’s required price, it notifies them.
Point to Ponder
Question
While users can search products directly on Amazon, why are they still interested in using the CamelCamelCamel service to search products that are ultimately fetched from Amazon?
There are a few reasons listed below for customers to use the C3 service instead of Amazon:
- Products searched from C3 are filtered and presented to the customer with attributes like current price, price timeline, the option to add the item to a watch list, etc.
- The customers will need to use C3 service to add the product to the watch list to get notified about price drops, even if they searched for it on Amazon.
- It has features that Amazon doesn’t offer, like displaying the top price drops directly to the user, which Amazon doesn’t offer.
- Multiple products can be added to the watch list in the same or different categories.
Moreover, this service can be used on top of Amazon as a browser extension to populate its features directly on the Amazon website, making it more useful and productive for customers.
The communication with PA-API of Amazon allows its customers to fetch data that can be used to advertise the products on different platforms, which increases sales. Although it offers a limited number of requests to each client, you can fetch multiple items (in the thousands) in a single request. C3 uses the same strategy to fetch the maximum number of items in each request to update the local database and to facilitate its users.
API Model for CamelCamelCamel Service
CamelCamelCamel API Design Evaluation and Latency Budget